m THETOlTEP STATE&ff^ENT.A^ 

Applicant: l%v# G. Collins Examine^:. Jeffrey S.. Smith 

Serial % • ,lJOy$iU30 Group Art Unit: 2624 

Filed: April % 2004 Docket"^: 2Q0400£i94 

title: GENBfTO^ 

DECLARATION OF RRIOR INVENTION UNDER 37 &E,R,S 1,131 
(SbBimiss.ioner for Patents 

E.bvBox mo 

Alexandria,: VA 22313-1450 
E)ef§r Sir^ad^: 

TK&Be^ 

present patehtapplieation. Xhe person rnaking this Decfera:(i^n isinventer EmMQ, <Miihs- 
Aceonipafiymg this Deelafafion is .Exhibit A to establisli^redMction to^ractke of the 
subject matter of the present patents io^^tlnijte^^ate^ prior to the : publication 

date o^^ajy ^ hereinafter 

Exhibit A (10: pages) include a HewlettrPadkard Company (H£?) Invention Disclosure, 
aud attaeliment submitted by the inyeftfor and received by the HP Legal D^iilttent ta 
February 12, 2Q04. In addiUoh; this invention rdiselosure^^^ 

2004; This myentibn disclosure was assigned HP Patent Disclosure No. 2004005 19. Tlie 

invention disclosure and attachment 

applicatioh. 

From this exhibit, it can be-seen that the subjecfcmatterqf the present patent 
application ws 12„2004of 
Allen, 

As a, person signing below, I hereby dectethat all statements made lierein of my 
own knowledge are true and; that all statementsmade on infonnation and belief are believed 
to be true; and fiptfis^ the Ibbwledge thai't willful false 

statements and the like so:mad<*are punishable by fine or imprisonment, or: both, under 
Section lapl of title 18 ^ (^odc, and tliat such willful false itafemenfe m# : 

jeopardize^ validity of the appiicatibn or any patcntissued thereon. 

Signed: MfrJ £ wA ^M-Bm^M 0 ^ 
David C. Collins (] 




invent 



EXHIBIT A paqe 1 of 10 



200400519: A Practical Implementati. 

L _^ ^„„ ~ . J.ll§ii§' , : W ^ £• 'ip ■ v 



Innovation Number 
200400519 




lllllS 



• ? ■ . i 



Title A Practical Implementation of Multipass Adaptive Wobulation 



Abstract Optimal subframe generation for wobulation can be implemented with an interative algorithm. 
This disclosure describes a practical method for implementing multiple passes of the 
wobulation subframe generation algorithm. 

Attachments ffi Adaptlve_KerneLWith Jtistory2.doc [188928 bytes] -| 
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The Information contained in this document is HP CONFIDENTIAL and may not be disclosed to others without prior authorization. 
Submit this disclosure to the HP Legal Department as soon as possible. No patent protection is possible until a patent application 
is authorized, prepared, and submitted to the Government. 



Title 



Abstract 



Projects 



Products 



A Practical implementation of Multipass Adaptive Wobulation 



Optimal subframe generation for wobulation can be implemented with 
an interative algorithm. This disclosure describes a practical method 
for implementing multiple passes of the wobulation subframe 
generation algorithm. 




Problems Solved 



Optimal subframe generation for wobulation is computationally 
complex. Niranjan at HP Labs has previously disclosed an iterative 
method for solving the optimal subframe generation problem. This 
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has been called the adaptive wobulation method. Running multiple 
iterations of the adaptive algorithm is expensive in terms of memory 
bandwidth, and is not practical for a consumer grade product. This 
disclosure describes a method for computing multiple passes of the 
adaptive algorithm with limited memory bandwidth requirements. 



Prior Solutions 



Two straight forward methods exist for optimal subframe generation 
using the adaptive method. The first method is to simply run the 
algorithm for multiple iterations. This results in the following steps: 1) 
Create the initial subframe 2) Generate a simulated image (i.e. merge 
the subframes together) 3) Calculate the error by subracting the 
simulated image from the original image 4) Feedback the error to 
update the subframes goto step 2 This method requires a great deal 
of memory bandwidth as the simulated image must be stored in 
memory, and the subframes must be stored in memory for each 
iteration. The second method is to apply the iterative algorithm on a 
portion of the image, and then calculate the final subframe value in a 
single step. Once a value have been calculated, the region of interest 
is shifted and a new value is calculated. This method requires a small 
amount of memory bandwidth, but many calculations are redundant, 
and the internal memory requirements of the calculation hardware are 
still quite large. The region of interest grows as the number of . 
interations grows. Thus, to date, only one pass of the adaptive 
algorithm has been seriously proposed. 
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Description 



The description is given in the attached document. In summary, the 
invention reduces the number of rows of data that are required for the 
multipass adaptive wobulation algorithm. This is accomplished by 
making a small compromise for some of the initial values, and by 
keeping track of the previous row of final subframe values. 
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Advantages 



This invention enables multipass wobulation to be implemented in an 
ASIC without increasing the memory bandwith requirements. Also, 
the amount of on chip memory is kept relatively small. The techniques 
in this invention can be applied in a straight forward fashion for the 
simplified center adaptive algorithm that was recently disclosed. 
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Adaptive Kernel With History 

The following explanation is assuming 4 position wobulation. Thus, four low resolution 
subframes must be generated subframes - one for each image position. All of the 
subframes are processed together at the same time, and the subframes are intertwined. 
Thus, every four pixel will correspond to a different subframe. The following diagram 
illustrates the idea. 




To compute the adaptive algorithm, subframes are computed, a simulated image ( the 
four subframes merged together) is computed, an error image is computed. The error is 
averaged, and then feedback into the subframes. This process is repeated iter atively, and 
within a few iterations, this method converges to the solution. The standard adaptive 
algorithm is briefly defined below: 



Simulation Kernel 



A 4 
4 4 W 

0 0 0 



Error Kernel 
0 (W 

U 4 4 
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Image 



4 4 



Iterative Algorithm 
Simulation = K s * Subframes 
Error = Image - Simulation 
Error aV g = K e * Error 
Subframes = Subframes + ax Error avg 



In order to calculate the subframe value for the a single pixel a 4x4 ROI is required for 
the one iteration. The diagram below illustrates the required pixel values. 
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ROT for Error Averaging 


























ROT for Initial Values 


















ROT for Simulation 





For two iterations, a 6x6 ROI is required. 









































ROI for 2 
Iterations 





































































For 3 iterations an ROI of 8x8 is needed, and in general fdr n iterations an ROI of 
(2n+2)x(2n+2) is required. 

Consider 3 iterations of the adaptive algorithm. To compute the optimal adaptive 
algorithm, an 8x8 ROI is required. 
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ROI for 3 
Iterations 



Gray Pixels Only 
Required for Initial Value 



The bottom row of the ROI is only required to generate the initial guess for the pixel 
values that are at least 2 pixels away from the primary pixel of interest. There is very 
little impact in eliminating the last row from the ROI. This just implies that for some of 
the pixels a simplified calculation will have to be used for the initial subframe value. 
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To generate the subframe values, the region of interest is positioned such that the pixel of 
interest is located over each pixel location. Typically, the ROI starts in the upper left 
corner, and the image is processed in a raster format Thus, the first row of subframe 
values are calculated, followed by the second, and then the third row of data. The 
following image illustrates how the ROI is moved across the image and the subframe 
values are generated. 
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Since the image processing is typically done in a raster pattern. Many of the final 
subframe values will already have their final values computed. The diagram below 
illustrates the pixel locations with final values. 
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"Final values calculafed: 



ROI for 3 
Iterations 




Bottom row ignored 
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Now if we look at the calculations we can see that instead of using three pixel rows of 
image data above the pixel of interest, we can substitute the final calculated values 
instead. Thus, we can compute the 3 -pass adaptive algorithm by using 1 row of the 
previously calculated subframe values along with 4 rows of the original image. This 
results in the following ROI: 
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tl values calculated : 



ROI for 3 Iterations with 
History (Final Values) 



Gray [Pixels Only \ 
Required for Initial Value 



Bottom row ignored 



This same approach can be used for a variety of iterative or adaptive algorithms. 
Essentially two main simplifications have been done. First, instead of using a number of 
image rows above the pixel of interest, only one row is used, and this row contains the 
calculated subframe values and not the original image values. The second simplification 
comes from truncating the kernel in heighth, but not in width. It should also be obvious 
that these techniques can be applied to achieve more than 3 iterations. Three iterations 
was used for the example because it highlights the novelties of the invention, and three is 
about the right number of iterations to achieve almost all of the benefits of the iterative 
algorithm. 



The benefits of this algorithm are best appreciated by an ASIC implementation. With 
this method the ASIC could be designed to include enough onboard memory to store the 
original image rows and one row of history (calculated subframe values). The image 
would be processed as it enters the ASIC. Only the final subframe values would need to 
be stored in external memory, and then read out at the proper time. Thus the memory 
bandwidth only has to support the writing and reading of a full frame of data. Also, as 
the kernel grows in height, more memory would be required on the ASIC. This explains 
why the goal was to reduce the number of rows needed by the ROL 
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In summary, the 3-pass adaptive kernel with history can be implemented by a relatively 
simply algorithm. The following steps are performed once per pixel location. 



Algorithm Steps 

1 Calculate Adaptive C4 (Initial 
Guess) 

2 Calculate Sim C4 (Convolve with 
Ks) 

3 Calculate Error C4 (Convolve with 
Ke) 

4 Calculate Sim C3 

5 Calculate Error C3 

6 Calculate Adaptive C3 (x = x + a 
Error ) 

7 Calculate Sim C2 

8 Calculate Error C2 

9 Calculate Adaptive C2 

10 Calculate Sim Ci 

11 Calculate Error Ci 
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Pixel of 
Interest 
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Adaptive 
Values 



Note: 

C5 is Column 5 
C4 is Column 4 
Etc... 



And for the truly inspired, the 3 pass adaptive algorithm can be implement with the 
following C++ code. The same approach has also been applied to the simplified center 
adaptive algorithm: 

unsigned char Adaptive StandardKernel :: Calculate 
( unsigned char finalO, 

unsigned char imagel, 

unsigned char image2, 

unsigned char image3, 

unsigned char image4 

) 

{ '* 

int temp; 

int x,y; . // temporary values to explicitly reduce the number of calculations 

shiftValuesO; 

finalO_5M - finalO; 

image 1_5M - imagel; 

image2_5M « image 2; 

image 3_5M = image 3; 

image4_5M « image4; 

// calculate guess for column 4 -> average of 4 image pixels 

x = imagel_4M+imagel_5M; 

y - image2_4M+image2_5M; 

guessl_4M = (x+y)»2; 

x » image 3_4M+ image 3_5M; 

guess2_4M = (x+y)»2; 

y » image4_4M+image4_5M; 

guess3_4M » (x+y)»2; 

guess 4_4M = y»l; 

// compute sim column 4 

x = finalO_3M+finalO_4M; 

y « guess l_3M+guessl_4M; 

int siml_4 = x+y; 

x = guess2_3M+guess2_4M; 

int sim2_4 = x+y; 
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y = guess3_3M+guess3_4M; 

int sim3_4 « x+y; 
x - guess4_3M+guess4_4M; 

int sim4_4 = x+y/ 

int errl_4 = (imagel_4M«2) - siml_ 
int err2_4 = (image2_4M«2) - sim2] 
int err3_4 - (image3_4M«2) - sim3~ 
int err4_4 = (image4_4M«2) - sim4~ 

// compute sim column 3 
x 

y 



4; 
4/ 
4/ 
4; 



// column 4 



int siml_3 
x 

int sim2_3 

y 

int sim3_3 - x+y; 



finalO_2M+finalO_3M; 
guessl_2M+guessl_3M; 
x+y; // column 3 
guess2_2M+guess2_3M; 
x+y; 

« guess 3_2M+guess3_3M; 



gues s4_2M+guess4_3M; 
int sim4_3 = x+y; 

int errl_3 = (imagel_3M«2) - siml 
int err2_3 - (image2_3M«2) - sim2~ 
int err3_3 = ( image 3_3M«2) - sim3~ 
int err4_3 = (image4_3M«2) - sim4~ 



3; 
3; 
3; 
3; 



// column 3 



// compute guess column 3 

// divide by 4 for average error, 

x - errl_3+errl_4; 

y - err2_3+err2_4; 

temp = x+y; 

temp = (temp * alphalM) » 6; 

temp = temp + guess 1_3M; 

guess 1_3M = max (0, min (temp, 255) ) ; 

x - err3_3+err3_4; 
temp - x+y; 

temp = (temp * alphalM) » 6; 
temp = temp + guess2_3M; 
guess2_3M « max (0, min (temp, 255) ) ; 

y «= err4_3+err4_4; 
temp = x+y; 

temp - (temp * alphalM) » 6; 
temp - temp + guess 3_3M; 
guess3_3M - max (0, min (temp, 255) ) ; 

temp « y; 

temp = (temp * alphalM) » 5; 
temp - temp + guess4,_3M; 
guess4_3M « max (0, min (temp, 255) ) ; 



A for simulation, & 4 for numerator 

// guess 1_3 

// clip value 

// guess 2_3 

// clip value 

// guess 3_3 

// clip value 

// guess 4_3 

// by 5 since x is not available 

// clip value 



X 




y 




int 


siml_2 = ; 


X 


= i 


int 


sim2_2 « ; 


y 


a ( 


int 


sim3_2 = : 


X 


» ( 


int 


sim4_2 « : 


int 


errl_2 = 


int 


err2_2 » 


int 


err3_2 = 


int 


err4_2 = 


X 


- errl I 


y 


- err2 : 


temp « x+y; 


temp «■ ( temp 



x+y; // column 2 
guess2_lM+guess2_2M; 
x+y; 

guess3_lM+guess3_2M; 
x+y; 

guess 4_lM+guess4_2M; 
x+y; 

( image 1_2M«2) - siml_ 
( image 2_2M«2) - sim2~ 
( image 3_2M«2) - sim3] 
(image4_2M«2) - sim4* 



2; 
2; 
2; 
2; 



// column 2 



// guess 1_2 
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temp = temp + guessl_2M; 

guessl_2M = max (0, min (temp, 255) ) ; // clip value 

x = err3_2+err3_3; 

temp = x+y; // guess 2_2 

temp = (temp * alpha2M> » 6; 
temp = temp + guess2_2M; 

guess2_2M = max (0, min (temp, 255) ) ; // clip value 

y = err4_2+err4_3; 

temp « x+y; // guess 3_2 

temp « (temp * alpha2M)»6; 
temp = temp + guess3_2M; 

guess3_2M » max (0, min (temp, 255) ) ; /> clip value 

temp « y; // guess 4_2 
temp » (temp * alpha2M)»5; 
temp » temp + guess 4_2M; 

guess4_2M » max (0, min (temp, 255) ) / // clip value 



int siml_l 
x 

int sim2_l 

y 

int sim3_l 
x 

int sim4_l 

int errl_l 
int err2_l 
int err3_l 
int err4 1 



finalO_OM+finalO_lM; 
guess l_0M+guessl_lM; 
x+y; // column 1 
guess2_0M+guess2_lM; 
x+y; 

guess3_0M+guess3_lM; 
x+y; 

guess4_0M+guess4JLM; 
x+y; 

v 

= (imagel_lM«2) - siml_l; 
= (image2_lM«2) - sim2_l; 
- (image3_lM«2) - sim3_l; 
= ( image 4_1M«2) - sim4_l; 



//' column 1 



x = errl_l+errl_2; 
y - err2^1+err2_2; 

temp - x+y;~ // guess 1_1 

temp « (temp * alpha3M)»6; 
temp = temp + guessl_lM; 

guesslJLM = max (0, min (temp, 255) ) ; // clip value 

x = err3_l+err3_2; 

temp - x+y; // guess 2JL 

temp - (temp * alpha3M)»6; 
temp = temp + guess 2_1M; 

guess2_lM = max (0, min (temp, 255) ) ; // clip value 

y « err4_l+err4_2; 

temp - x+y;" // guess 3_1 

temp « (temp * alpha3M)»6; 
temp = temp + guess3_lM; 

guess3_lM » max (0, min (temp, 255) ) ; // clip value 

temp = y; // guess 4_1 

temp = (temp * alpha3M)»5; 
temp = temp + guess4_lM; 

guess4_lM = max (0, min (temp, 255) ) ; // clip value 



return guessl_0M; 



